Systems and methods for improved processing of message queries

ABSTRACT

A method of processing messages executes at a computing device having one or more processors and memory. The memory stores one or more programs configured for execution by the one or more processors. A first message for a user is analyzed for a structured content element. When found, content from a data source distinct from the message is obtained by executing an associated action. A first message display state is formed for the message comprising a message notification and the content. A messaging application user interface, comprising an electronic message list with a plurality of objects, is updated. Responsive to selection of a first object of the plurality of objects, the first object representing the first message, the user is enabled to toggle the first message display state between first and second display states, the second display state differing from the first by providing direct access to the first message.

This application is a continuation of U.S. application Ser. No.16/160,234, filed Oct. 15, 2018, which is a continuation of U.S.application Ser. No. 14/473,571, filed Aug. 29, 2014, now U.S. Pat. No.10,104,028, which are incorporated herein by reference in theirentirety.

TECHNICAL FIELD

The disclosure relates generally to processing electronic messages, andmore specifically to providing multiple display states for objectsrepresenting messages.

BACKGROUND

Electronic messaging applications, such as email, convey various typesof information to a recipient such as messages to and from others. Inconventional messaging applications, it is not always possible toascertain the relative importance of each message in an incoming boxwithout first reviewing their full content. Moreover, even forrelatively important messages, it is often necessary to open themessage, that is review their full content, in order to derive necessaryor useful information. Thus, a recipient of messages falls into arecurring pattern of not only checking for new messages, but alsomanually opening each message to obtain needed or useful information.What are needed in the art are systems and methods for assisting arecipient with obtaining useful information from messages more quicklyand with fewer manual operations.

SUMMARY

Disclosed implementations address the above deficiencies and otherproblems associated with reviewing messages. In accordance with someimplementations, a method of processing messages executes at a computingdevice having one or more processors and memory. The memory stores oneor more programs configured for execution by the one or more processors.A message, addressed to a user, is analyzed for the presence of a firststructured content element. Nonlimiting examples of structured contentelements include those that identify the message as regarding an airlineflight, those that identify the message as regarding a hotelreservation, those that identify the message as regarding a restaurantreservation, those that identify the message as regarding a purchase,those that identify the message as regarding a ticketed event, thosethat identify the message as including an attachment, those thatidentify the message as including an article or video, those thatidentify the message as regarding an invitation in which an RSVP isrequested.

When the first structured content element is found in the message, amethod is performed that comprises executing an action associated withthe first structured content element thereby obtaining content from afirst data source that is distinct from the message. In someembodiments, the first structured content element is an identificationof an airline flight within the message and the data source is currentflight information for the airline flight.

Typically, the message is analyzed by comparing the message, or portionsof the message to multiple different structured content elements todetermine if the message has any such structured content element. Whenthe message is deemed to have or match one of the different structuredcontent elements, action based on this particular different structuredcontent element is taken in order to obtain content from a first datasource that is distinct from the message. In some instances, thesemultiple different structured content elements are predetermined, storedcontent elements that are system defined, as opposed to user defined.

In some such embodiments, the first structured content element is in aplurality of structured content elements and the analyzing of themessage comprises analyzing the message for the presence of eachstructured content element in the plurality of elements. When a matchbetween the message and one of the different structured content elementsis found, an action associated with that one structured content elementis performed thereby obtaining content from a data source that isdistinct from the message.

In some embodiments, a first display state of the object representingthe message is formed when the first structured content element is foundin the message. This first display state comprises a combination of anotification of the message and the content found based on the presenceof the first structured content element.

In some embodiments, each structured content element in the plurality ofcontent elements is associated with a different image stored in a seconddata source that is distinct from both the first data source and themessage. In such embodiments, forming a first display state comprisesincluding the image associated with the first structured content element(the structured element in the plurality of content elements that wasdeemed to have been found in the message) in the first display state ofthe object representing the message along with the combination of thenotification of the message and the content. In an example of such anembodiment is one in which the message pertains to a flight reservation,the first data source is real time flight information for a flight inthe flight reservation, and the second data source includes an image ofthe destination of a flight in the flight reservation.

Responsive to identifying the presence of the first structured contentelement in the message, a user interface for a messaging application isupdated. The user interface comprises an electronic message list in anelectronic message list panel. The electronic message list comprises aplurality of objects. Each respective object in the plurality of objectsis visually distinct in the electronic message list. Each respectiveobject in at least a subset of the plurality of objects comprises one ormore electronic messages in a plurality of messages addressed to theuser.

Responsive to user selection of a first object of the plurality ofobjects in the electronic message list panel, the first objectrepresenting at least the message, the system facilitates user initiatedtoggling of a display state of the object representing the messagebetween the first display state and a second display state. The seconddisplay state differs from the first display state by providing directaccess to the message. That is, whereas the first display state providesa notification of the message in addition to information that is not inthe message, it is necessary to go to the second display state in orderto see the message, or a direct link to the message.

In some embodiments, a task is associated with the first structuredcontent element and the forming the first display state comprisesincluding information for performing the task in the first display stateof the object representing the message along with the combination of thenotification of the message and the content. In some embodiments, theinformation for performing the task is in the form of a uniform resourcelocator. In some embodiments, the first structured content elementidentifies an airline flight and the task is an option to cancel theflight. In some embodiments, the first structured content elementidentifies a hotel or restaurant reservation and the task is an optionto cancel the hotel or restaurant reservation, an option to confirm thehotel or restaurant reservation, an option to modify the hotel orrestaurant reservation, or an option to view a map of a location of anaddress associated with the hotel or restaurant reservation. In somesuch embodiments, the forming the first display state comprisesincluding a date associated with the hotel or restaurant reservation,where the date is retrieved from the message.

In some embodiments, the first structured content element is an airlineflight with a flight origin and a flight destination, and the methodfurther comprises obtaining a first image associated with the flightorigin and a second image associated with the flight destination from asecond data source that is distinct from (i) the message and (ii) thefirst data source and the forming the first display state comprisesincluding the first image and the second image in the first displaystate of the object representing the message along with the combinationof the notification of the message and the content. An example ofcontent from the first data source in such embodiments is real timeflight information.

In some embodiments, the first structured content element identifies apurchase of a product and the content is a picture of the product. Insome embodiments, the first structured content element identifies apurchase of a product and the content is an estimated delivery date ofthe product. In some such embodiments, a task is associated with thefirst structured content element and the forming the first display statecomprises including information for performing the task in the firstdisplay state of the object representing the message along with thecombination of the notification of the message and the content. Forexample, in some embodiments, the task is viewing an original order forthe product or viewing tracking details for a delivery of the product.

In some embodiments, the first structured content element identifies aticketed event and the content is a picture associated with the ticketedevent. In some such embodiments, a task is associated with the firststructured content element and the forming the first display statecomprises including information for performing the task in the firstdisplay state of the object representing the message along with thecombination of the notification of the message and the content. In someembodiments, the task is viewing tickets for the ticketed events orviewing a map associated with a location of the ticketed event.

In some embodiments, the first structured content element is a link toan image and the content is a rendering of the image. In someembodiments, the first structured content element is a link to a videoand the content is a frame from the video.

In some embodiments, the first object exclusively represents themessage. That is, in such embodiments, the first object does notrepresent two or more messages. In other embodiments, the first objectrepresents a plurality of messages and the object includes the firstdisplay state of the object representing the message.

Another aspect of the present disclosure provides a computing devicecomprising one or more processors, memory, and one or more programsstored in the memory. The one or more programs are configured forexecution by the one or more processors. The one or more programscomprise instructions for analyzing a message, addressed to a user, forthe presence of a first structured content element. When the firststructured content element is found in the message, a method isperformed that comprises (i) executing an action associated with thefirst structured content element thereby obtaining content from a firstdata source that is distinct from the message, and (ii) forming a firstdisplay state of the object representing the message. The first displaystate of the object representing the message comprises a combination ofa notification of the message and the content. Responsive to identifyingthe presence of the first structured content element in the message, auser interface for a messaging application is updated. The userinterface comprises an electronic message list in an electronic messagelist panel. The electronic message list comprises a plurality ofobjects. Each respective object in the plurality of objects is visuallydistinct in the electronic message list. Each respective object in atleast a subset of the plurality of objects comprises one or moreelectronic messages in a plurality of messages addressed to the user.Responsive to user selection of a first object of the plurality ofobjects in the electronic message list panel, the first objectrepresenting at least the message, user initiated toggling of a displaystate of the object representing the message between first and seconddisplay states is facilitated. The second display state differs from thefirst display state by providing direct access to the message.

Another aspect of the present disclosure provides a non-transitorycomputer readable storage medium storing one or more programs configuredfor execution by a computing device having one or more processors andmemory. The one or more programs comprise instructions for analyzing amessage, addressed to a user, for the presence of a first structuredcontent element. When the first structured content element is found inthe message, a method is performed that comprises (i) executing anaction associated with the first structured content element therebyobtaining content from a first data source that is distinct from themessage, and (ii) forming a first display state of the objectrepresenting the message. The first display state of the objectrepresenting the message comprises a combination of a notification ofthe message and the content. Responsive to identifying the presence ofthe first structured content element in the message, a user interfacefor a messaging application is updated. The user interface comprises anelectronic message list in an electronic message list panel. Theelectronic message list comprises a plurality of objects. Eachrespective object in the plurality of objects is visually distinct inthe electronic message list. Each respective object in at least a subsetof the plurality of objects comprises one or more electronic messages ina plurality of messages addressed to the user. Responsive to userselection of a first object of the plurality of objects in theelectronic message list panel, the first object representing at leastthe message, user initiated toggling of a display state of the objectrepresenting the message between first and second display states isfacilitated. The second display state differs from the first displaystate by providing direct access to the message.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the aforementioned implementations of theinvention as well as additional implementations thereof, referenceshould be made to the Description of Implementations below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1 illustrates a context in which some implementations operate.

FIG. 2 is a block diagram of a client computing device according to someimplementations.

FIG. 3 is a block diagram of a server according to some implementations.

FIGS. 4A-4K illustrate aspects of a user interface providing multipledisplay states for objects representing messages in accordance with someimplementations.

FIGS. 5A-5D provide a flowchart of a process for providing multipledisplay states for objects representing messages according to someimplementations.

Reference will now be made in detail to implementations, examples ofwhich are illustrated in the accompanying drawings. In the followingdetailed description, numerous specific details are set forth in orderto provide a thorough understanding of the present invention. However,it will be apparent to one of ordinary skill in the art that the presentinvention may be practiced without these specific details.

DESCRIPTION OF IMPLEMENTATIONS

FIG. 1 is a block diagram that illustrates the major components of someimplementations. The various client devices 102 (also identified hereinas computing devices) and servers 300 in a server system 110 communicateover one or more networks 108 (such as the Internet). In variousembodiments, a client device 102 is a smart phone, a tablet computer, alaptop computer, a mobile computing device, a computing device of avehicle of a user (e.g., an in-vehicle communications system, anin-vehicle entertainment system, an in-vehicle navigation system), awearable apparatus of the user that includes a computing device (e.g., awatch of the user having a computing device, glasses of the user havinga computing device, a wearable music player), desktop computer, or othercomputing device that has access to a communication network 108 and canrun an electronic messaging application 106. In some implementations,the messaging application runs within a web browser 104.

In some implementations, the server system 110 consists of a singleserver 300. More commonly, the server system 110 includes a plurality ofservers 300. In some implementations, the servers 300 are connected byan internal communication network of bus 124. The server system 110includes one or more web servers 112, which receive requests from users(e.g., from client devices 102) and return appropriate information,resources, links, and so on. In some implementations, the server system110 includes one or more application servers 114, which provide variousapplications, such as a messaging application 106. The server system 110typically includes one or more databases 116, which store informationsuch as web pages, a user list 118, various user information 120 (e.g.,user names and encrypted passwords, user preferences, and so on), aplurality of predefined candidate task reminders 122, first data source126 and second data source 128.

FIG. 2 is a block diagram illustrating a client device 102 that a useruses to access a messaging application 106. A client device is alsoreferred to as a computing device, which may be a tablet computer, alaptop computer, a smart phone, a desktop computer, a PDA, or othercomputing device than can run an electronic messaging application 106and has access to a communication network 108. A client device 102typically includes one or more processing units (CPUs) 202 for executingmodules, programs, or instructions stored in memory 214 and therebyperforming processing operations; one or more network or othercommunications interfaces 204; memory 214; and one or more communicationbuses 212 for interconnecting these components. In some embodiments, thecommunication buses 212 include circuitry (sometimes called a chipset)that interconnects and controls communications between systemcomponents. A client device 102 includes a user interface 206 comprisinga display device 208 and one or more input devices or mechanisms 210. Insome implementations, the input device/mechanism includes a keyboard anda mouse; in some implementations, the input device/mechanism includes a“soft” keyboard, which is displayed as needed on the display device 208,enabling a user to “press keys” that appear on the display 208.

In some implementations, the memory 214 includes high-speed randomaccess memory, such as DRAM, SRAM, DDR RAM or other random access solidstate memory devices. In some implementations, memory 214 includesnon-volatile memory, such as one or more magnetic disk storage devices,optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. In some implementations, thememory 214 includes one or more storage devices remotely located fromthe CPU(s) 202. The memory 214, or alternately the non-volatile memorydevice(s) within memory 214, comprises a non-transitory computerreadable storage medium. In some implementations, the memory 214, or thecomputer readable storage medium of memory 214, stores the followingprograms, modules, and data structures, or a subset thereof:

-   -   an operating system 216, which includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a communications module 218, which is used for connecting the        client device 102 to other computers and devices via the one or        more communication network interfaces 204 (wired or wireless)        and one or more communication networks 112, such as the        Internet, other wide area networks, local area networks,        metropolitan area networks, and so on;    -   a display module 220, which receives input from the one or more        input devices 210, and generates user interface elements for        display on the display device 208;    -   a web browser 104, which enables a user to communicate over a        network 108 (such as the Internet) with remote computers or        devices;    -   a messaging application 106, which enables the user to send and        receive electronic messages. In some implementations, the        messaging application is an email application. In some        implementations, the messaging application is an instant        messaging application. In some implementations, the messaging        application 106 runs within the web browser 104 as illustrated        in FIG. 1. In some implementations, the messaging application        106 runs independently of a web browser 104 (e.g., a desktop        application). An example of an electronic messaging application        is illustrated below in FIGS. 4A-4K; and    -   application data 222, which is used by the messaging application        106. The application data comprises messages 224 (e.g., email        messages or instant messages). In some embodiments, the        application data 222 includes application configuration data        226, such as user preferences, user history, geographic        information about the user, or the state of configuration        options.

Each of the above identified executable modules, applications, or setsof procedures may be stored in one or more of the previously mentionedmemory devices and corresponds to a set of instructions for performing afunction described above. The above identified modules or programs(e.g., sets of instructions) need not be implemented as separatesoftware programs, procedures, or modules, and thus various subsets ofthese modules may be combined or otherwise re-arranged in variousimplementations. In some implementations, the memory 214 stores a subsetof the modules and data structures identified above. Furthermore, insome embodiments, the memory 214 stores additional modules or datastructures not described above.

Although FIG. 2 shows a client device 102, FIG. 2 is intended more as afunctional description of the various features that may be presentrather than as a structural schematic of the implementations describedherein. In practice, and as recognized by those of ordinary skill in theart, items shown separately could be combined and some items could beseparated.

FIG. 3 is a block diagram illustrating a server 300 that is used in aserver system 110 in accordance with some embodiments. A typical serversystem includes many individual servers 300, which, in some instances,numbers in the hundreds or thousands. A server 300 typically includesone or more processing units (CPUs) 302 for executing modules, programs,or instructions stored in the memory 314 and thereby performingprocessing operations; one or more network or other communicationsinterfaces 304; memory 314; and one or more communication buses 312 forinterconnecting these components. In some embodiments, the communicationbuses 312 include circuitry (sometimes called a chipset) thatinterconnects and controls communications between system components. Insome implementations, a server 300 includes a user interface 306, whichmay include a display device 308 and one or more input devices 310, suchas a keyboard and a mouse.

In some implementations, the memory 314 includes high-speed randomaccess memory, such as DRAM, SRAM, DDR RAM or other random access solidstate memory devices. In some implementations, the memory 314 includesnon-volatile memory, such as one or more magnetic disk storage devices,optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. In some implementations, thememory 314 includes one or more storage devices remotely located fromthe CPU(s) 302. The memory 314, or alternately the non-volatile memorydevice(s) within memory 314, comprises a non-transitory computerreadable storage medium. In some implementations, the memory 314, or thecomputer readable storage medium of memory 314, stores the followingprograms, modules, and data structures, or a subset thereof:

-   -   an operating system 316, which includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a communications module 318, which is used for connecting the        server 300 to other computers via the one or more communication        network interfaces 304 (wired or wireless), an internal network        or bus 124, or other communication networks 108, such as the        Internet, other wide area networks, local area networks,        metropolitan area networks, and so on;    -   an optional display module 320, which receives input from one or        more input devices 310, and generates user interface elements        for display on a display device 308;    -   one or more web servers 112, which receive requests from client        device 102, and returns responsive web pages, resources, or        links. In some implementations, each request is logged in the        database 116;    -   one or more application servers 114, which provide various        applications (such as an email or other electronic messaging        application) to client devices 102. In some instances,        applications are provided as a set of web pages, which are        delivered to client devices 102 and displayed in a web browser        104. The web pages are delivered as needed or requested. In some        instances, an application is delivered to a client device 102 as        a download, which is installed and run from the client device        102 outside of a web browser 104;    -   one or more databases 116, which store various data used by the        modules or programs identified above. In some implementations,        the database 116 includes a list of authorized users 118 (e.g.,        including user names, encrypted passwords, and other relevant        information about each user). The database 116 also stores user        specific data 120 that is used by one or more of the        applications provided by the application server. For example,        some implementations store the electronic messages 224 for each        user. For example, some implementations store objects 320 for        each user. As another example, some implementations store user        profiles 340 associated with users. In some implementations, the        database 116 includes candidate structured content elements 122.        In some implementations, the database 116 includes first data        source 126 and second data source 128.

Candidate task reminders 122 and/or 228 are in any form suitable forreminding a user of a task. Moreover, it will be appreciated that FIGS.2 and 3 collectively represent the server 300 and the client device 102as including predefined candidate task reminders. In someimplementations, only the server 300 stores predefined candidate taskreminders. In some implementations, only the client device 102 storespredefined candidate task reminders. In some implementations, both theserver 300 and the client device 102 stores predefined candidate taskreminders. In some such implementations, the server 300 and the clientdevice 102 store the same predefined candidate task reminders. In somesuch implementations, the server 300 stores a first set of predefinedcandidate task reminders and the client device 102 stores a second setof predefined candidate task reminders, and there is no overlap betweenthe first set of predefined candidate task reminders and the second setof predefined candidate task reminders, partial overlap between thefirst set of predefined candidate task reminders and the second set ofpredefined candidate task reminders, or complete overlap between thefirst set of predefined candidate task reminders and the second set ofpredefined candidate task reminders. In one embodiment, the first set ofpredefined candidate task reminders is defined by a source distinct fromthe user, and the second set of predefined candidate task reminders isdefined by a particular user associated with a particular client device102.

Each of the above identified elements in FIG. 3 may be stored in one ormore of the previously mentioned memory devices. Each executableprogram, module, or procedure corresponds to a set of instructions forperforming a function described above. The above identified modules orprograms (e.g., sets of instructions) need not be implemented asseparate software programs, procedures or modules, and thus varioussubsets of these modules are combined or otherwise re-arranged invarious implementations. In some implementations, the memory 314 storesa subset of the modules and data structures identified above.Furthermore, in some embodiments, the memory 314 stores additionalmodules or data structures not described above.

Although FIG. 3 illustrates a server 300, FIG. 3 is intended more asfunctional illustration of the various features that may be present in aset of one or more servers rather than as a structural schematic of theimplementations described herein. In practice, and as recognized bythose of ordinary skill in the art, items shown separately could becombined and some items could be separated. The actual number of serversused to implement these features, and how features are allocated amongthem, will vary from one implementation to another, and depends in parton the amount of data traffic that the system must handle during peakusage periods as well as during average usage periods.

As illustrated in FIGS. 2 and 3, the functionality for an electronicmessaging application may be shared between a client device 102 and aserver system 110. In some implementations, after the electronicmessaging application is installed on a client device 102, the majorityof the subsequent processing occurs on the client device. In otherimplementations, the majority of the processing and data storage occursat a server 300, and the client device 102 uses a web browser 104 toview and interact with the data. One of skill in the art recognizes thatvarious allocations of functionality between the client device 102 andthe server system 110 are possible, and some implementations supportmultiple configurations (e.g., based on user selection).

FIG. 4A illustrates features of a messaging application that providemultiple display states for objects representing messages in accordancewith some implementations. FIG. 4A illustrates user interface 402, whichis displayed as part of a messaging application (e.g., an emailapplication). User interface depicts a plurality of objects in a listformat. In this example, each row in the list represents a separateobject. Examples of objects include messages, (e.g., messages 404, 408,and 414-424) and message clusters (e.g., 406, 410). Other forms ofobjects are possible too, such as tasks. The objects are all for a userassociated with the user interface 402. In some embodiments, individualmessages (e.g., messages 404, 408, and 414-424) are message threads,message conversations, or messages that are addressed to one or moreusers, including the user uniquely associated with the user interface402.

In FIG. 4A, the user interface 402 is, in particular, shown to compriseat least two objects that are message clusters, cluster 406 and cluster410. Cluster 406 is a group of electronic messages having a collectiveassociation, namely promotional material (e.g., advertisements forretail sales, newsletters from stores). As can be seen in FIG. 4A,cluster 406 comprises nine new electronic messages, and cluster 410comprises one new electronic message. In some embodiments, a newelectronic message refers to a message that has not been viewed, opened,acknowledged or read by the user of the electronic messagingapplication. Message clusters are disclosed further in U.S. patentapplication Ser. No. 14/145,005, entitled “Systems and Methods forThrottling Display of Electronic Messages,” filed Dec. 31, 2013, andU.S. patent application Ser. No. 14/139,205, entitled “Systems andMethods for Clustering Electronic Messages,” filed Dec. 23, 2013, eachof which is hereby incorporated by reference herein in its entirety.

Continuing to refer to FIG. 4A, a method of providing multiple displaystates for objects representing messages is disclosed in more detail. Amessage (not shown), addressed to a user, is analyzed for the presenceof a first structured content element. When the first structured contentelement is found in the message, an action associated with the firststructured content element is performed thereby obtaining content from afirst data source that is distinct from the message. For instance,consider the case in which the first structured content element is anidentification of an airline flight and the data source is currentflight information for the airline flight. When a message is determinedto have the first structured content element, the airline flight, thisstructured information is used as a basis to obtain the current flightinformation from a data source that is distinct from the message, suchas first data source 126. For instance, the airline flight from thefirst email is inserted into a search expression or structured querythat is used to search the data source for current information regardingthe flight. In one example, the first structured element refers toflight 1451 from SFO to ORD, and the expression 1451 from the message isused to obtain the current departure time for the flight from a datasource that is distinct from the message.

In some embodiments, when the first structured content element is foundin the message, an action associated with the first structured contentelement is performed thereby obtaining content from multiple datasources, such as the message that includes the first structured contentelement, one or more databases, and/or one or more other messagesreceived by the user. For instance, in some embodiments, informationfrom across multiple messages is obtained. As one example, informationis obtained from one message that pertains to an order confirmation andadditional information is obtained from another message that pertains toorder shipment. As another example, information is obtained from onemessage that pertains to a hotel reservation confirmation and additionalinformation is obtained from another message that pertains tocancellation of the hotel reservation.

When a structured content element is found in the message and contentreceived from the data source distinct from the message, a first displaystate of a first object representing at least the message is formed. Thefirst display state of the first object comprises a combination of anotification of the message and the content that was retrieved from adata source that is distinct from the message (e.g., the currentdeparture time for the flight). It will be appreciated that thenotification of the message is not the message itself, nor does itprovide direct access to the message. The user interface 402 is updatedto reflect the addition of the object representing the message. In thisexample, the message (not shown) included two flights, and thusinformation on departure time for both flights is provided in an object412 that represents the message. As FIG. 4A illustrates, the firstdisplay state of object 412, denoted state 412-1, includes a synopsis426 of each flight from the message, and information on departure time428 for both flights from the first data source that is distinct fromthe message. Also provided in the first display state of object 412 isan indication 434 of whether the flight to JFK is on time, which isobtained from, or derived from the information in, the first data sourcethat is distinct from the message (e.g., data source 126).

It will be appreciated that data can be obtained from any number of datasources distinct from the message to create the first display state ofthe object representing the message in addition to, or instead of,information from the message itself. Accordingly, in some embodiments,the identification of the first structured content element is used as abasis for obtaining content from more than one data source that isdistinct from the analyzed (first) message. For instance, in the messagethat forms the basis of creating object 412 of FIG. 4A, the structuredcontent element relates to two flights. Accordingly, current informationon flight status 428 is obtained from a first data source (e.g., firstdata source 126 of FIGS. 1 and 3) and a hero image 430 associated withthe designation of one of the flights, JFK, is obtained from a seconddata source (e.g., second data source 128 of FIGS. 1 and 3).

In some instances, a task is also inferred from the message and added tothe object representing the message. Ways to infer a task from anelectronic message are disclosed in U.S. patent application Ser. No.14/161,368, entitled “Identifying Tasks in Messages,” filed Jan. 22,2014, which is hereby incorporated by reference herein in its entirety.In some embodiments a message is processed in order to determine whetherthe message includes a task by determining, for each respectivepredefined candidate task in a plurality of predefined tasks, whetherthere is a match between the predefined candidate task and all or aportion of the respective message. In some embodiments, this matchbetween the predefined candidate task and all or a portion of therespective message includes natural language processing of the all or aportion of the respective message as disclosed in U.S. patentapplication Ser. No. 14/161,368, entitled “Identifying Tasks inMessages,” filed Jan. 22, 2014, which is hereby incorporated byreference herein in its entirety. In some embodiments, each predefinedcandidate task is a rule, heuristic, phrase, or other expression that ismatched or applied against respective messages addressed to a user(e.g., the incoming message) to see if there is a match between thepredefined candidate task and all or a portion of the respectivemessage. In some embodiments, a predefined candidate task in theplurality of candidate tasks is from a source other than (external to,independent of) the user. For instance, in some embodiments there is alibrary of standard or common predefined candidate task formats that areprovided for matching. When a match is deemed to have been found, theobject corresponding to the message that had a match is updated toinclude the task derived from the corresponding message. In somealternative embodiments, the task is determined from the identity of thestructured content element found in the message. Thus, in the case ofFIG. 4A, the task is facilitating flight check in 432. When the userclicks on or otherwise selects task affordance 432, information on howto check in to a flight referenced in the message, or a URL for checkinginto the flight referenced in the message is provided.

Responsive to user selection of object 412, user initiated toggling ofthe display state of the object 412 between the first display state(412-1) and a second display state (412-2) is facilitated. FIG. 4Aillustrates the first display state 412-1 of object 412 and FIG. 4Billustrates the second display state 412-2 of object 412. The seconddisplay state differs from the first display state by providing directaccess 450 to the message. Further, as FIGS. 4A and 4B illustrate, thefirst display state 412-1 is a collapsed display state and the seconddisplay state 412-2 is an expanded display state. In other words, thefirst display state 412-1 provides less information than the seconddisplay state 412-2. More generally, in typical embodiments, theinformation provided by the second display state of the objectrepresenting the message is different than the information provided bythe first display state of the object representing the message.

As FIG. 4B illustrates, the second display state of object 412, denotedstate 412-2, includes terminal and gate information 452 of each flightin the message. Such information is from a source that is distinct fromthe message. The second display state 412-2 also has information ondeparture time 428 for each such flight from a data source that isdistinct from the message. Also provided in the second display state412-2 of object 412 is an indication 434 as to whether the flight to JFKis on time. Further, the second display state 412-2 includes updatedinformation on the flight departure time 428 of the flight from SFO andthe flight from ORD and a hero image 430 associated with the finaldestination, JFK, that is obtained from a data source that is distinctfrom the underlying message (e.g., second data source 128 of FIGS. 1 and3). Display state 412-2 also displays a confirmation number 460 for theflights. In addition to being in an expanded format, display state 412-2of FIG. 4B differs from display state 412-1 of FIG. 4A for object 412 byincluding either the text of, or a direct link 450 to the message thatincluded the first structured content element and that is represented byobject 412.

In some embodiments, the expanded display state 412-2 of FIG. 4B isreached by selection of any point within the display space occupied bydisplay state 412-1 in FIG. 4A other than affordance 432. In someembodiments, special action, such as swiping or double clicking ortapping within the screen area occupied by display state 412-1 isperformed to reach display state 412-2. In some embodiments it isreached by selection of hero image 430 of FIG. 4A. In some embodimentsit is reached by selection of a designated “expand” affordance that isnot shown in FIG. 4A.

Turning once again to FIG. 4B it is seen that, responsive to userselection of the collapsed display state 412-1, for example by any ofthe ways described above, the collapsed display state 412-2 of object412 is replaced with an expanded state 412-2 by pushing any objects inthe list of objects that are above the object 412 (e.g., objects 404,406, 408, and 410) up and pushing any objects in the list of objectsthat are below the object 412 (e.g., objects 414, 416, 418, 420, 422,and 424) down. In alternative embodiments, the expanded state 412-2 ofobject 412 occupies the entire portion of user interface 402 that isdevoted to listing objects, such that no other object are listed whenthe expanded state 412-2 of object 412 is displayed.

In some embodiments, the collapsed display state 412-1 of FIG. 4A, isreached from the expanded display state 412-2 of FIG. 4B by selection ofany point within the display space occupied by display state 412-2 inFIG. 4B other than affordances 432 and 450. In some embodiments, specialaction, such as swiping or double clicking or tapping within the displaystate 412-2 is performed to reach display state 412-1. In someembodiments it is reached by selection of hero image 430 of FIG. 4B. Insome embodiments it is reached by selection of a designated “collapse”affordance that is not shown in FIG. 4B. Responsive to user designatedselection to return to the collapsed state 412-1, the expanded state412-2 of object 412 is once again replaced with the original collapsedstate 412-1 of the object by pushing any objects in the list of objectsthat are above object 412 (e.g., objects 404, 406, 408, and 410) downand pushing any objects in the list of objects that are below object 412(e.g., objects 414, 416, 418, 420, 422, and 424) up in the userinterface.

FIG. 4C illustrates the collapsed state 602-1 of the first objectrepresenting a message that included a structured content element forflights in accordance with another embodiment. Accordingly, displaystate 602-1 is an alternative to display state 412-1 of FIG. 4A. Displaystate 602-1 includes information on departure time 428 for both flightsfrom the first data source that is distinct from the message. Alsoprovided in the display state 602-1 is an indication 434 of whether theflight to JFK is on time, which is obtained from, or derived from theinformation in, the first data source that is distinct from the message(e.g., data source 126). Display state 602-1 also includes gateinformation 452 and a flight confirmation number 460.

FIG. 4D illustrates the collapsed state 604-1 of the first objectrepresenting a message that included a structured content element forflights in accordance with still another embodiment. Accordingly,display state 604-1 is an alternative to display state 412-1 of FIG. 4A.Display state 604-1 includes task affordance 432. When the user selectstask affordance 432, information on how to check in to a flightreferenced in the message, or a URL for checking into the flightreferenced in the message, is provided. Display state 604-furtherincludes a hero image 430-1 representing the flight origination city anda separate hero image 430-2 representing flight destination city, bothof which are taken from a source that is distinct from the message.Display state 604-1 further includes updated flight departure andarrival times 470 for the flight that is obtained from a source that isindependent of the message.

FIG. 4E illustrates a collapsed display state 606-1 of an objectrepresenting a message (not shown), addressed to a user, in which thefirst structured content element pertains to a hotel reservation. Whenthe first structured content element is found in the message, an actionassociated with the first structured content element is performedthereby obtaining content from a first data source that is distinct fromthe message. For instance, in the case of FIG. 4E, the first structuredcontent element is an identification of a hotel reservation and the datasource is an image of the hotel taken from a source distinct from themessage. Here, when the message is determined to have the firststructured content element, the hotel reservation, this structuredinformation (the name of the hotel) is used as a basis to obtain theimage of the hotel from a data source that is distinct from the message,such as first data source 126. Accordingly, as described before, when astructured content element is found in a message and content receivedfrom the data source distinct from the message, a first display state ofa first object representing at least the message is formed. The firstdisplay state of the first object comprises a combination of anotification of the message and the content that was retrieved from adata source that is distinct from the message (e.g., the picture of thehotel). The user interface 402 is updated to reflect the addition of theobject representing the message. As FIG. 4E illustrates, the firstdisplay state of object 606, denoted state 606-1, includes the name ofthe hotel 482, the dates 484 covered by the reservation, and a link 486to a map of the hotel. Further a picture of the hotel 480 from a sourcedistinct from the message is provided. FIG. 4F illustrates acorresponding second expanded display state 606-2 of object 606 in whichadditional information such as the hotel address 682, the message 490,and tasks associated with hotel reservations (viewing a map of the hotel486 and modifying the hotel reservation 488) are depicted. In someembodiments, such tasks are identified based on the identity of thestructured content item identified in the first email (a hotelreservation).

Prior examples provided object display states for messages that were notincorporated into clusters. As indicated before, it is possible for amessage to be clustered in to a cluster using, for example thetechniques disclosed in U.S. patent application Ser. No. 14/145,005,entitled “Systems and Methods for Throttling Display of ElectronicMessages,” filed Dec. 31, 2013, and U.S. patent application Ser. No.14/139,205, entitled “Systems and Methods for Clustering ElectronicMessages,” filed Dec. 23, 2013, each of which is hereby incorporated byreference herein in its entirety. FIG. 4G illustrates an example inwhich the message has been clustered into a purchase cluster 610. Thatis, the message is deemed to not only include a first structured contentelement that warrants generating an object with collapsed and expandeddisplay states as in other embodiments, it also is deemed to belong topurchase cluster 610. FIG. 4G represents the use case in which themessage has been clustered into the purchase cluster 610 but has notbeen read yet. When this is the case, the collapsed display state 612-1of an object 612 representing the message is displayed in the purchasecluster object 610. The collapsed display state 612-1 includes thestandard components of the collapsed display state of an objectrepresenting a message that includes a first structured content itempertaining to purchases, in accordance with this example, including thenumber of items purchase 614, expected delivery date 616, an image 618of an item purchased (where the image is optionally from a sourcedistinct from the message), the task of tracking 620 the package thatcontains the purchase, and an indication 622 of the state of shipment ofthe package (again from a source distinct from the message).

FIG. 4H illustrates the corresponding expanded display state of object612. The expanded display state 612-2 includes the standard componentsof the expanded display state of an object representing a message thatincludes a first structured content item pertaining to purchases, inaccordance with this example, including the number of items purchase614, expected delivery date 616, an images 618 of items purchased (wheresuch image are optionally from a source distinct from the message), thetask of tracking 620 the package that contains the purchase, anindication 622 of the state of shipment of the package (again from asource distinct from the message). The expanded display state 612-2further includes a view order task 630 and the original underlyingmessage 632. A user can return to the corresponding collapsed displaystate 612-1 of FIG. 4G from expanded state 612-2 of FIG. 4H by selectingaffordance 634.

FIG. 4I illustrates a collapsed display state 640-1 of an objectrepresenting a message (not shown), addressed to a user, in which thefirst structured content element pertained to a restaurant reservation.When the first structured content element is found in the message, anaction associated with the first structured content element is performedthereby obtaining content from a first data source that is distinct fromthe message. For instance, in the case of FIG. 4I, the first structuredcontent element is an identification of a restaurant reservation and thedata source is an image of the restaurant taken from a source distinctfrom the message. Here, when the message is determined to have the firststructured content element, the restaurant reservation, this structuredinformation (the name of the restaurant) is used as a basis to obtainthe image of the restaurant from a data source that is distinct from themessage, such as first data source 126. Accordingly, as describedbefore, when a structured content element is found in a message andcontent received from the data source distinct from the message, a firstdisplay state of a first object representing at least the message isformed. The first display state of the first object comprises acombination of a notification of the message and the content that wasretrieved from a data source that is distinct from the message (e.g.,the picture of the restaurant). The user interface 402 is updated toreflect the addition of the object representing the message. As FIG. 4Iillustrates, the first display state of object 640, denoted state 640-1,includes the name of the restaurant 642, the date and time of thereservation 644 covered by the reservation, and a link 646 to URL thatallows modification of the reservation. Further, a picture of therestaurant 648 from a source distinct from the message is provided.

FIG. 4J illustrates a corresponding second expanded display state 640-2of object 640 in which additional information such as the restaurantaddress 670, the message 672, and tasks associated with restaurantreservations (viewing a map of the restaurant 674 and modifying thehotel reservation 676) is illustrated. In some embodiments, such tasksare identified based on the identity of the structured content itemidentified in the first email (a restaurant reservation).

FIG. 4K illustrates a collapsed display state 680-1 of an objectrepresenting a message (not shown), addressed to a user, in which thefirst structured content element pertains to a ticket event. When thefirst structured content element is found in the message, an actionassociated with the first structured content element is performedthereby obtaining content from a first data source that is distinct fromthe message. For instance, in the case of FIG. 4K, the first structuredcontent element is an identification of a ticketed event and the datasource is an image of the restaurant taken from a source distinct fromthe message. Here, when the message is determined to have the firststructured content element, the ticketed event, this structuredinformation (the ticketed event) is used as a basis to obtain the imageof the arena hosting the ticketed event from a data source that isdistinct from the message, such as first data source 126. Accordingly,as described before, when a structured content element is found in amessage and content received from the data source distinct from themessage, a first display state of a first object representing at leastthe message is formed. The first display state of the first objectcomprises a combination of a notification of the message and the contentthat was retrieved from a data source that is distinct from the message(e.g., the picture of the arena or other forum hosting the ticketedevent). The user interface 402 is updated to reflect the addition of theobject representing the message. As FIG. 4K illustrates, the first(collapsed) display state of the object 680, denoted state 680-1,includes the name of the ticketed event 682, the name 684 of the arenaor other forum hosting the ticketed event, the date and time 686 of theticketed event date, the option to perform the task 688 of viewing theticket, the number of tickets purchase 690, and the task 692 of viewinga map of the arena or other forum hosting the ticketed event.

Referring to FIG. 2, a display device 208 is inherently limited in size.Accordingly, the number of objects displayed in the user interface 402is necessarily limited. Scrolling may be necessary for a user to get toa desired message. If the number of messages in the message list issmall enough, all of the messages may be able to be displayed.

FIGS. 5A-5D provide a flowchart of a process 500, performed by acomputing device, for providing multiple display states for objectsrepresenting messages (502). The method is performed (504) at acomputing device 102 and/or 110 having one or more processors andmemory. The memory stores (504) one or more programs configured forexecution by the one or more processors.

A message, addressed to a user, is analyzed for the presence of a firststructured content element. When the first structured content element isfound in the message, a method is performed that comprises (i) executingan action associated with the first structured content element therebyobtaining content from a first data source that is distinct from themessage and (ii) forming a first display state of a first objectrepresenting at least the message, where the first display state of thefirst object comprises a combination of a notification of the messageand the content (506). In some embodiments, the first structured contentelement is an identification of an airline flight and the first datasource is current flight information for the airline flight 508.

In some embodiments, the first structured content element is in aplurality of candidate structured content elements and the analyzing ofthe message comprises analyzing the message for the presence of eachcandidate structured content element in the plurality of candidatestructured content elements thereby identifying the first structuredcontent element as a match between all or a portion of the message and acandidate structured content element of the plurality of candidatestructured content elements (510). In some such embodiments, eachcandidate structured content element in the plurality of candidatestructured content elements is associated with a different image storedin a second data source that is distinct from the message, and theforming the first display state comprises including the image associatedwith the first structured content element in the first display state ofthe first object along with the combination of the notification of themessage and the content (512).

In some embodiments, there is a task is associated with the firststructured content element and the forming the first display statecomprises including information for performing the task in the firstdisplay state of the first object along with the combination of thenotification of the message and the content (514). In some embodiments,the information for performing the task is in the form of a uniformresource locator (516). In some embodiments, the first structuredcontent element identifies an airline flight and the task is an optionto cancel the flight (518). In some embodiments, the first structuredcontent element identifies a hotel or restaurant reservation and thetask is an option to cancel the hotel or restaurant reservation, anoption to confirm the hotel or restaurant reservation, an option tomodify the hotel or restaurant reservation, or an option to view a mapof a location of an address associated with the hotel or restaurantreservation (520). In some such embodiments, the forming the firstdisplay state comprises including a date associated with the hotel orrestaurant reservation, wherein the date is retrieved from the message(522).

In some embodiments, the first structured content element is an airlineflight with a flight origin and a flight destination and the methodfurther comprises obtaining a first image associated with the flightorigin and a second image associated with the flight destination from asecond data source that is distinct from (i) the message and (ii) thefirst data source; and the first display state includes the first imageand the second image along with the combination of the notification ofthe message and the content (524).

In some embodiments, the first structured content element identifies apurchase of a product and the content is a picture of the product (526).

In some embodiments, the first structured content element identifies apurchase of a product and the content is an estimated delivery date ofthe product (528). In some such embodiments, there is a task isassociated with the first structured content element and the forming thefirst display state comprises including information for performing thetask (e.g., viewing an original order for the product or viewingtracking details for a delivery of the product) in the first displaystate of the first object along with the combination of the notificationof the message and the content (530).

In some embodiments, the first structured content element identifies aticketed event and the content is a picture associated with the ticketedevent (532). In some such embodiments, a task is associated with thefirst structured content element and the forming the first display statecomprises includes information for performing the task (e.g., viewingtickets for the ticketed events or viewing a map associated with alocation of the ticketed event) in the first display state of themessage along with the combination of the notification of the messageand the content (534).

In some embodiments, the first structured content element is a link toan image and the content is a rendering of the image (536). In someembodiments, the first structured content element is a link to a videoand the content is a frame from the video (538). In some embodiments,the first object exclusively represents the message (540). In someembodiments, the first object represents a plurality of messages (542).

Responsive to identifying the presence of the first structured contentelement in the message, a user interface for a messaging application isupdated. The user interface comprises an electronic message list in anelectronic message list panel. The electronic message list comprises aplurality of objects including the first object. Each respective objectin the plurality of objects is visually distinct in the electronicmessage list. Further, each respective object in at least a subset ofthe plurality of objects comprises one or more electronic messages in aplurality of messages addressed to the user (544). Responsive to userselection of the first object, user initiated toggling of a displaystate of the first object is facilitated between the first display stateand a second display state, where the second display state differs fromthe first display state by providing direct access to the message andwhere the first display state is a collapsed display state and thesecond display state is an expanded display state (546).

The terminology used in the description of the invention herein is forthe purpose of describing particular implementations only and is notintended to be limiting of the invention. As used in the description ofthe invention and the appended claims, the singular forms “a,” “an,” and“the” are intended to include the plural forms as well, unless thecontext clearly indicates otherwise. It will also be understood that theterm “and/or” as used herein refers to and encompasses any and allpossible combinations of one or more of the associated listed items. Itwill be further understood that the terms “comprises” and/or“comprising,” when used in this specification, specify the presence ofstated features, steps, operations, elements, and/or components, but donot preclude the presence or addition of one or more other features,steps, operations, elements, components, and/or groups thereof.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific implementations. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theimplementations described herein were chosen and described in order tobest explain the principles of the invention and its practicalapplications, to thereby enable others skilled in the art to bestutilize the invention and various implementations with variousmodifications as are suited to the particular use contemplated.

What is claimed is:
 1. A method comprising: determining, by one or moreprocessors of a computing device, whether a first message, addressed toa user, includes a first structured content element that includes anindication of a purchase; responsive to determining that the firstmessage includes the first structured content element: forming, by theone or more processors, a first display state of a first objectrepresenting at least the first message, wherein the first display stateof the first object comprises a combination of a notification of thefirst message and an indication of a state of shipment of a package froma first data source distinct from the first message; and updating, bythe one or more processors, a user interface for a messaging applicationto include the first object in an electronic message list in anelectronic message list panel, wherein the electronic message listcomprises a plurality of objects including the first object; andresponsive to user selection of the first object, toggling, by the oneor more processors, a display state of the first object between thefirst display state and a second display state, wherein the seconddisplay state includes the indication of the state of the shipment ofthe package from the first data source and the first message, whereinthe second display state differs from the first display state by atleast in part providing direct access to the first message, wherein thefirst display state is a collapsed display state for the first objectpresented within the messaging application, and the second display stateis an expanded display state for the first object presented within themessaging application, wherein the first display state and the seconddisplay state are each configured to be used by the messagingapplication for displaying the first object, and wherein the indicationof the purchase provided by the second display state of the objectincludes a view order task and an original underlying message pertainingto the purchase that are not provided in the first display state of theobject.
 2. The method of claim 1, wherein the first structured contentelement is in a plurality of candidate structured content elements, andwherein determining whether the first message includes the firststructured content element comprises identifying the first structuredcontent element as a match between all or a portion of the first messageand a candidate structured content element of the plurality of candidatestructured content elements.
 3. The method of claim 2, wherein eachcandidate structured content element in the plurality of candidatestructured content elements is associated with a respective image storedin a second data source that is distinct from the first message, andwherein forming the first display state further comprises including therespective image associated with the first structured content element inthe first display state of the first object along with the combinationof the notification of the first message and the state of shipment ofthe package.
 4. The method of claim 1, wherein a task is associated withthe first structured content element, and wherein the forming the firstdisplay state comprises including information for performing the task inthe first display state of the first object along with the combinationof the notification of the first message and the state of shipment ofthe package.
 5. The method of claim 4, wherein the information forperforming the task is in the form of a uniform resource locator.
 6. Themethod of claim 4, wherein the task is one or more of viewing anoriginal order for a product in the package or viewing tracking detailsfor a delivery of the package.
 7. The method of claim 1, wherein thefirst structured content element further comprises a purchase of anitem, wherein the method further comprises obtaining a first imageassociated with the item purchased, and wherein forming the firstdisplay state comprises including the first image in the first displaystate of the first object along with the combination of the notificationof the first message and the indication of the state of shipment of thepackage.
 8. The method of claim 1, wherein the first object exclusivelyrepresents the first message.
 9. The method of claim 1, wherein thefirst object represents a plurality of messages.
 10. The method of claim1, wherein the first data source that is distinct from the first messageis a second message addressed to the user.
 11. A computing device,comprising: one or more processors; memory; and one or more programsstored in the memory configured for execution by the one or moreprocessors, the one or more programs comprising instructions for:determining whether a first message, addressed to a user, includes afirst structured content element that includes an indication of apurchase; responsive to determining that the first message includes thefirst structured content element: forming, by the one or moreprocessors, a first display state of a first object representing atleast the first message, wherein the first display state of the firstobject comprises a combination of a notification of the first messageand an indication of a state of shipment of a package from a first datasource distinct from the first message; and updating, by the one or moreprocessors, a user interface for a messaging application to include thefirst object in an electronic message list in an electronic message listpanel, wherein the electronic message list comprises a plurality ofobjects including the first object; and responsive to user selection ofthe first object, toggling, by the one or more processors, a displaystate of the first object between the first display state and a seconddisplay state, wherein the second display state includes the indicationof the state of the shipment of the package from the first data sourceand the first message, wherein the second display state differs from thefirst display state by at least in part providing direct access to thefirst message, wherein the first display state is a collapsed displaystate for the first object presented within the messaging application,and the second display state is an expanded display state for the firstobject presented within the messaging application, wherein the firstdisplay state and the second display state are each configured to beused by the messaging application for displaying the first object, andwherein the indication of purchase provided by the second display stateof the object includes a view order task and an original underlyingmessage pertaining to the purchase that are not provided in the firstdisplay state of the object.
 12. The computing device of claim 11,wherein the first structured content element is in a plurality ofcandidate structured content elements, and wherein the one or moreprograms comprise instructions for identifying the first structuredcontent element as a match between all or a portion of the first messageand a candidate structured content element of the plurality of candidatestructured content elements.
 13. The computing device of claim 12,wherein each candidate structured content element in the plurality ofcandidate structured content elements is associated with a respectiveimage stored in a second data source that is distinct from the firstmessage, and wherein the one or more programs comprise instructions forincluding the respective image associated with the first structuredcontent element in the first display state of the first object alongwith the combination of the notification of the first message and thestate of shipment of the package.
 14. The computing device of claim 12,wherein a task is associated with the first structured content element,and wherein the one or more programs comprise instructions for includinginformation for performing the task in the first display state of thefirst object along with the combination of the notification of the firstmessage and the state of shipment of the package.
 15. The computingdevice of claim 14, wherein the information for performing the task isin the form of a uniform resource locator.
 16. The computing device ofclaim 14, wherein the task is one or more of viewing an original orderfor a product in the package or viewing tracking details for a deliveryof the package.
 17. The computing device of claim 11, wherein the firststructured content element further comprises a purchase of an item, andwherein the one or more programs further comprise instructions for:obtaining a first image associated with the item purchased, andincluding the first image in the first display state of the first objectalong with the combination of the notification of the first message andthe indication of the state of shipment of the package.
 18. Thecomputing device of claim 11, wherein the first object exclusivelyrepresents the first message.
 19. The computing of claim 11, wherein thefirst data source that is distinct from the first message is a secondmessage addressed to the user.
 20. A computer readable medium storingone or more programs configured for execution by a computing devicehaving one or more processors and memory, the one or more programscomprising instructions for: determining, by one or more processors of acomputing device, whether a first message, addressed to a user, includesa first structured content element that includes an indication of apurchase; responsive to determining that the first message includes thefirst structured content element: forming, by the one or moreprocessors, a first display state of a first object representing atleast the first message, wherein the first display state of the firstobject comprises a combination of a notification of the first messageand an indication of a state of shipment of a package from a first datasource distinct from the first message; and updating, by the one or moreprocessors, a user interface for a messaging application to include thefirst object in an electronic message list in an electronic message listpanel, wherein the electronic message list comprises a plurality ofobjects including the first object; and responsive to user selection ofthe first object, toggling, by the one or more processors, a displaystate of the first object between the first display state and a seconddisplay state, wherein the second display state includes the indicationof the state of the shipment of the package from the first data sourceand the first message, wherein the second display state differs from thefirst display state by at least in part providing direct access to thefirst message, wherein the first display state is a collapsed displaystate for the first object presented within the messaging application,and the second display state is an expanded display state for the firstobject presented within the messaging application, wherein the firstdisplay state and the second display state are each configured to beused by the messaging application for displaying the first object, andwherein the indication of purchase provided by the second display stateof the object includes a view order task and an original underlyingmessage pertaining to the purchase that are not provided in the firstdisplay state of the object